Improvements and Generalizations of Stochastic Knapsack and Multi-Armed Bandit Algorithms: Extended Abstract

The celebrated multi-armed bandit (MAB) problem, originating from the work of Gittins et al. [GGW89], presumes a condition on the arms called the martingale assumption. Recently, A. Gupta et al. obtained an LP-based 1 48 -approximation for the problem with the martingale assumption removed [GKMR11]. We improve the algorithm to a 4 27 -approximation, with simpler analysis. Our algorithm also generalizes to the case of MAB superprocesses with (stochastic) multi-period actions. This generalization captures the explore-exploit budgeted learning framework introduced by Guha and Munagala [GM07a, GM07b]. Also, we obtain a tight ( 12 − ε)-approximation for the variant where preemption (playing an arm, switching to another arm, then coming back to the first arm) is not allowed. This contains the stochastic knapsack problem of Dean, Goemans, and Vondrák [DGV08] with correlated rewards, for both the cancellation and no cancellation cases, improving the 1 16 and 1 8 approximations of [GKMR11], respectively. Our algorithm samples probabilities from an exponential-sized dynamic programming solution, whose existence is guaranteed by an LP projection argument. We hope this technique can also be applied to other dynamic programming problems which can be projected down onto a small LP. ∗willma353@gmail.com, Operations Research Center, Massachusetts Institute of Technology. Supported in part by the NSERC PGS-D Award, NSF grant CCF-1115849, and ONR grants N00014-11-1-0053 and N00014-11-1-0056.